Multimodal Haptic Feedback for Virtual Collisions Combining Vibrotactile and Electrical Muscle Stimulation.

In this paper, we explore the effects of multimodal haptic feedback combining vibrotactile and electrical muscle stimulation (EMS) on expressing virtual collisions. We first present a wearable multimodal haptic device capable of generating both mechanical vibration and EMS stimuli. The two types of haptic stimulus are combined into a haptic rendering method that conveys improved virtual collision sensations. This multimodal rendering method highlights the strengths of each modality while compensating for mutual weaknesses. The multimodal rendering method is compared in subjective quality with two unimodal methods (vibration only and EMS only) by a user study. Experimental results demonstrate that our multimodal feedback method can elicit more realistic, enjoyable, expressive, and preferable user experiences.

Haptic Interactions Subject to Variable Latency.

Model-Mediated Teleoperation (MMT) between a haptic device and a remote or virtual environment uses a local model of the environment to compensate for latency of communication. MMT is often case-specific, and requires underlying latency distributions to be known. We propose a novel approach - which we refer to as the DelayRIM - which uses the time-stepping aspect of a Reduced Interface Model for the environment to render an up-to-date force to the haptic device from the delayed information. RIM is applicable to any physical or virtual system, and the DelayRIM itself makes no underlying assumption about the latency distribution. We show that for realistic variable delays, the DelayRIM improves transparency compared to other methods for a virtual drone bilateral teleoperation scenario.

Intuitive Cell Manipulation Microscope System with Haptic Device for Intracytoplasmic Sperm Injection Simplification.

In recent years, the demand for effective intracytoplasmic sperm injection (ICSI) for the treatment of male infertility has increased. The ICSI operation is complicated as it involves delicate organs and requires a high level of skill. Several cell manipulation systems that do not require such skills have been proposed; notably, several automated methods are available for cell rotation. However, these methods are unfeasible for the delicate ICSI medical procedure because of safety issues. Thus, this study proposes a microscopic system that enables intuitive micropipette manipulation using a haptic device that safely and efficiently performs the entire ICSI procedure. The proposed system switches between field-of-view expansion and three-dimensional image presentation to present images according to the operational stage. In addition, the system enables intuitive pipette manipulation using a haptic device. Experiments were conducted on microbeads instead of oocytes. The results confirmed that the time required for the experimental task was improved by 52.6%, and the injection error was improved by 75.3% compared to those observed in the conventional system.

iCub3 avatar system: Enabling remote fully immersive embodiment of humanoid robots.

We present an avatar system designed to facilitate the embodiment of humanoid robots by human operators, validated through iCub3, a humanoid developed at the Istituto Italiano di Tecnologia. More precisely, the paper makes two contributions: First, we present the humanoid iCub3 as a robotic avatar that integrates the latest significant improvements after about 15 years of development of the iCub series. Second, we present a versatile avatar system enabling humans to embody humanoid robots encompassing aspects such as locomotion, manipulation, voice, and facial expressions with comprehensive sensory feedback including visual, auditory, haptic, weight, and touch modalities. We validated the system by implementing several avatar architecture instances, each tailored to specific requirements. First, we evaluated the optimized architecture for verbal, nonverbal, and physical interactions with a remote recipient. This testing involved the operator in Genoa and the avatar in the Biennale di Venezia, Venice-about 290 kilometers away-thus allowing the operator to visit the Italian art exhibition remotely. Second, we evaluated the optimized architecture for recipient physical collaboration and public engagement on stage, live, at the We Make Future show, a prominent world digital innovation festival. In this instance, the operator was situated in Genoa while the avatar operated in Rimini-about 300 kilometers away-interacting with a recipient who entrusted the avatar with a payload to carry on stage before an audience of approximately 2000 spectators. Third, we present the architecture implemented by the iCub Team for the All Nippon Airways (ANA) Avatar XPrize competition.

Human Factors in Sensory Feedback for Haptic Gloves.

Manual therapy training requires close proximity between the clinical teacher and students, which limits the training of people in remote and rural regions. Video-based online training can provide visual but not tactile information, which is also essential for manual therapies. This project describes the development and testing of an inexpensive sensor glove developed using commercially available sensors, suitable for monitoring the shape and force applied by the hand of a person delivering a spinal manipulation. Its focus was the development of software to provide the human user with tactile information that is usually acquired intuitively in face-to-face teaching. Though rigorous assessment of the glove's application showed errors at low levels of force in actual force measurement and interpretation by users, these errors were reduced at higher levels of force. Trainers of spinal manipulation reported the device to be very useful and suitable for the purpose. We conclude that this glove has the potential for being used for online training of students.Clinical Impact: The outcome of this study shows the feasibility of developing an inexpensive haptic glove using proprietary software for online training of students of manual therapy.

Distally-referred surface electrical nerve stimulation (DR-SENS) for haptic feedback.

Objective.This study's objective is to understand distally-referred surface electrical nerve stimulation (DR-SENS) and evaluates the effects of electrode placement, polarity, and stimulation intensity on the location of elicited sensations in non-disabled individuals.Approach.A two-phased human experiment was used to characterize DR-SENS. In Experiment One, we explored 182 electrode combinations to identify a subset of electrode position combinations that would be most likely to elicit distally-referred sensations isolated to the index finger without discomfort. In Experiment Two, we further examined this subset of electrode combinations to determine the effect of stimulation intensity and electrode position on perceived sensation location. Stimulation thresholds were evaluated using parameter estimation by sequential testing and sensation locations were characterized using psychometric intensity tests.Main Results.We found that electrode positions distal to the wrist can consistently evoke distally referred sensations with no significant polarity dependency. The finger-palm combination had the most occurrences of distal sensations, and the different variations of this combination did not have a significant effect on sensation location. Increasing stimulation intensity significantly expanded the area of the sensation, moved the most distal sensation distally, and moved the vertical centroid proximally. Also, a large anodic-leading electrode at the elbow mitigated all sensation at the anodic-leading electrode site while using symmetric stimulation waveforms. Furthermore, this study showed that the most intense sensation for a given percept can be distally referred. Lastly, for each participant, at least one of the finger-palm combinations evaluated in this study worked at both perception threshold and maximum comfortable stimulation intensities.Significance.These findings show that a non-invasive surface electrical stimulation charge modulated haptic interface can be used to elicit distally-referred sensations on non-disabled users. Furthermore, these results inform the design of novel haptic interfaces and other applications of surface electrical stimulation based haptic feedback on electrodes positioned distally from the wrist.

Design and Assessment of a Bimanual Haptic Epidural Needle Insertion Simulator.

The case experience of anesthesiologists is one of the leading causes of accidental dural punctures and failed epidurals-the most common complications of epidural analgesia used for pain relief during delivery. We designed a bimanual haptic simulator to train anesthesiologists and optimize epidural analgesia skill acquisition. We present an assessment study conducted with 22 anesthesiologists of different competency levels from several Israeli hospitals. Our simulator emulates the forces applied to the epidural (Touhy) needle, held by one hand, and those applied to the Loss of Resistance (LOR) syringe, held by the other one. The resistance is calculated based on a model of the epidural region layers parameterized by the weight of the patient. We measured the movements of both haptic devices and quantified the results' rate (success, failed epidurals, and dural punctures), insertion strategies, and the participants' answers to questionnaires about their perception of the simulation realism. We demonstrated good construct validity by showing that the simulator can distinguish between real-life novices and experts. Face and content validity were examined by studying users' impressions regarding the simulator's realism and fulfillment of purpose. We found differences in strategies between different level anesthesiologists, and suggest trainee-based instruction in advanced training stages.

Easy-To-Wear Auxetic SMA Knot-Architecture for Spatiotemporal and Multimodal Haptic Feedbacks.

Wearable haptic interfaces prioritize user comfort, but also value the ability to provide diverse feedback patterns for immersive interactions with the virtual or augmented reality. Here, to provide both comfort and diverse tactile feedback, an easy-to-wear and multimodal wearable haptic auxetic fabric (WHAF) is prepared by knotting shape-memory alloy wires into an auxetic-structured fabric. This unique meta-design allows the WHAF to completely expand and contract in 3D, providing superior size-fitting and shape-fitting capabilities. Additionally, a microscale thin layer of Parylene is coated on the surface to create electrically separated zones within the WHAF, featuring zone-specified actuation for conveying diverse spatiotemporal information to users with using the WHAF alone. Depending on the body part it is worn on, the WHAF conveys either cutaneous or kinesthetic feedback, thus, working as a multimodal wearable haptic interface. As a result, when worn on the forearm, the WHAF intuitively provides spatiotemporal information to users during hands-free navigation and teleoperation in virtual reality, and when worn on the elbow, the WHAF guides users to reach the desired elbow flexion, like a personal exercise advisor.

Toward Designing Haptic Displays for Desired Touch Targets: A Study of User Expectation for Haptic Properties via Crowdsourcing.

Things that people desire to touch in daily life are known to be limited to a number of specific targets (e.g., cats). The utilization of haptic displays to provide the experience of touching such desired targets is expected to enhance people's quality of life. However, it is currently unclear which haptic properties (e.g., hardness and weight) of desired targets should be rendered with haptic displays, and how they should be rendered. To address these issues, we conducted an experiment with 600 Japanese participants via crowdsourcing. Among the 600 participants, we identified potential users of haptic displays and analyzed their responses for each target. For each desired target, we identified the haptic properties in relation to which a "need for consistency" was felt by potential users between their expectations and actual impressions during touching. We also identified the haptic properties in relation to which a "biased impression" was held by potential users for each target. For example, potential users responded that cats were soft and that the actual impression of softness during touching needed to be consistent with their impression. Our results provide insights into the design of haptic displays for realizing desired touch experiences.

Macro-Fiber Composite-Based Tactors for Haptic Applications.

Haptic technology is a critical component of human-computer interfaces. Traditional haptic actuators are often unable to provide the broad frequency range and latency that is required in many advanced applications. To address this problem, we propose a new type of tactor based on macro-fiber composites (MFCs), composites of piezoelectric fibers. We propose a physics-based model for the actuation of the tactors, calibrated and validated through experiments. As our tactors are intended for haptic applications, we consider the role of skin on their response, an aspect seldom analyzed in the literature. In our experiments, we simulate the presence of the skin with a rubber membrane in contact with the tactor, with varying pre-stretch, mimicking different indentations of the tactor on the skin. The MFC-based tactor can always generate vibration amplitudes higher than skin discrimination thresholds, over the range of frequencies of interest for haptics, with a latency much smaller than traditional actuators. We theoretically investigate the effect of the skin on tactor vibrations, highlighting the individual roles of skin stiffness and damping and their combined effect across a series of pre-stretches. Our tactor shows promise in haptic applications, including assistive technologies and real-time feedback systems for training, safety, and monitoring.

Bimanual Intravenous Needle Insertion Simulation Using Nonhomogeneous Haptic Device Integrated into Mixed Reality.

In this study, we developed a new haptic-mixed reality intravenous (HMR-IV) needle insertion simulation system, providing a bimanual haptic interface integrated into a mixed reality system with programmable variabilities considering real clinical environments. The system was designed for nursing students or healthcare professionals to practice IV needle insertion into a virtual arm with unlimited attempts under various changing insertion conditions (e.g., skin: color, texture, stiffness, friction; vein: size, shape, location depth, stiffness, friction). To achieve accurate hand-eye coordination under dynamic mixed reality scenarios, two different haptic devices (Dexmo and Geomagic Touch) and a standalone mixed reality system (HoloLens 2) were integrated and synchronized through multistep calibration for different coordinate systems (real world, virtual world, mixed reality world, haptic interface world, HoloLens camera). In addition, force-profile-based haptic rendering proposed in this study was able to successfully mimic the real tactile feeling of IV needle insertion. Further, a global hand-tracking method, combining two depth sensors (HoloLens and Leap Motion), was developed to accurately track a haptic glove and simulate grasping a virtual hand with force feedback. We conducted an evaluation study with 20 participants (9 experts and 11 novices) to measure the usability of the HMR-IV simulation system with user performance under various insertion conditions. The quantitative results from our own metric and qualitative results from the NASA Task Load Index demonstrate the usability of our system.

Representational Similarity Analysis for Tracking Neural Correlates of Haptic Learning on a Multimodal Device.

A goal of wearable haptic devices has been to enable haptic communication, where individuals learn to map information typically processed visually or aurally to haptic cues via a process of cross-modal associative learning. Neural correlates have been used to evaluate haptic perception and may provide a more objective approach to assess association performance than more commonly used behavioral measures of performance. In this article, we examine Representational Similarity Analysis (RSA) of electroencephalography (EEG) as a framework to evaluate how the neural representation of multifeatured haptic cues changes with association training. We focus on the first phase of cross-modal associative learning, perception of multimodal cues. A participant learned to map phonemes to multimodal haptic cues, and EEG data were acquired before and after training to create neural representational spaces that were compared to theoretical models. Our perceptual model showed better correlations to the neural representational space before training, while the feature-based model showed better correlations with the post-training data. These results suggest that training may lead to a sharpening of the sensory response to haptic cues. Our results show promise that an EEG-RSA approach can capture a shift in the representational space of cues, as a means to track haptic learning.

Perceived realism of haptic rendering methods for bimanual high force tasks: original and replication study.

Realistic haptic feedback is a key for virtual reality applications in order to transition from solely procedural training to motor-skill training. Currently, haptic feedback is mostly used in low-force medical procedures in dentistry, laparoscopy, arthroscopy and alike. However, joint replacement procedures at hip, knee or shoulder, require the simulation of high-forces in order to enable motor-skill training. In this work a prototype of a haptic device capable of delivering double the force (35 N to 70 N) of state-of-the-art devices is used to examine the four most common haptic rendering methods (penalty-, impulse-, constraint-, rigid body-based haptic rendering) in three bimanual tasks (contact, rotation, uniaxial transition with increasing forces from 30 to 60 N) regarding their capabilities to provide a realistic haptic feedback. In order to provide baseline data, a worst-case scenario of a steel/steel interaction was chosen. The participants needed to compare a real steel/steel interaction with a simulated one. In order to substantiate our results, we replicated the study using the same study protocol and experimental setup at another laboratory. The results of the original study and the replication study deliver almost identical results. We found that certain investigated haptic rendering method are likely able to deliver a realistic sensation for bone-cartilage/steel contact but not for steel/steel contact. Whilst no clear best haptic rendering method emerged, penalty-based haptic rendering performed worst. For simulating high force bimanual tasks, we recommend a mixed implementation approach of using impulse-based haptic rendering for simulating contacts and combine it with constraint or rigid body-based haptic rendering for rotational and translational movements.

Cable-Driven Haptic Interface With Movable Bases Achieving Maximum Workspace and Isotropic Force Exertion.

Haptic interactions play an essential role in education to enhance learning efficiency; however, haptic information for virtual educational content remains lacking. This article proposes a planar cable-driven haptic interface with movable bases that can display isotropic force feedback with maximum workspace extension on a commercial screen display. A generalized kinematic and static analysis of the cable-driven mechanism is derived by considering movable pulleys. Based on the analyses, a system including movable bases is designed and controlled to maximize the workspace subject to isotropic force exertion for the target screen area. The proposed system is evaluated experimentally as a haptic interface represented by the workspace, isotropic force-feedback range, bandwidth, Z-width, and user experiment. The results indicate that the proposed system can maximize workspace to the target rectangular area and exert isotropic force up to 94.0% of the theoretical computed one within the workspace.

HaptGlove-Untethered Pneumatic Glove for Multimode Haptic Feedback in Reality-Virtuality Continuum.

Novel haptics technologies are urgently needed to bridge the gap between entirely physical world and fully digital environment to render a more realistic and immersive human-computer interaction. Current virtual reality (VR) haptic gloves either deliver limited haptic feedback or are bulky and heavy. The authors develop a haptic glove or HaptGlove, an untethered and lightweight pneumatic glove, that allows users to "physically" interact in a VR environment and enables both kinesthetic and cutaneous sensations naturally and realistically. Integrated with five pairs of haptic feedback modules and fiber sensors, HaptGlove provides variable stiffness force feedback and fingertip force and vibration feedback, allowing users to touch, press, grasp, squeeze, and pull various virtual objects and feel the dynamic haptic changes. Significant improvements in VR realism and immersion are observed in a user study with participants achieving 78.9% accuracy in sorting six virtual balls of different stiffnesses. Importantly, HaptGlove facilitates VR training, education, entertainment, and socialization in a reality-virtuality continuum.

EEG-Based Machine Learning Models to Evaluate Haptic Delay: Should We Label Data Based on Self-Reporting or Physical Stimulation?

Reliable haptic interfaces augment human-computer interaction via simulated tactile and kinesthetic feedback. As haptic technologies advance, user experience evaluation becomes more crucial. Conventionally, self-reporting is used to evaluate haptic experiences; however, it could be inconsistent or imprecise due to human error. A promising alternative is using neurocognitive methods with machine or deep learning models to evaluate the human haptic experience. Machine and deep learning models can be trained on Electroencephalography (EEG) data labeled based on self-report or actual physical stimulation. As the literature lacks a systematic study on which approach is more robust, we develop a visuo-haptic task to answer this question by examining an important haptic experience, namely, haptic delay. EEG is recorded during the experiment, and participants report whether they detected a delay in the haptic modality through self-report. Four machine/deep learning models were trained twice on the EEG data using the two labeling methods. Models trained with labels from the physical stimuli significantly outperformed those trained with self-reporting labels. Although this finding holds true for one particular haptic experience (haptic delay), it cannot be extrapolated to others; rather, it suggests that EEG data labeling plays a prominent role in evaluating the haptic experience through neurocognitive methods.

PixeLite: A Thin and Wearable High Bandwidth Electroadhesive Haptic Array.

We present PixeLite, a novel haptic device that produces distributed lateral forces on the fingerpad. PixeLite is 0.15 mm thick, weighs 1.00 g, and consists of a 4×4 array of electroadhesive brakes ("pucks") that are each 1.5 mm in diameter and spaced 2.5 mm apart. The array is worn on the fingertip and slid across an electrically grounded countersurface. It can produce perceivable excitation up to 500 Hz. When a puck is activated at 150 V at 5 Hz, friction variation against the countersurface causes displacements of 627 ± 59 µm. The displacement amplitude decreases as frequency increases, and at 150 Hz is 47 ± 6 µm. The stiffness of the finger, however, causes a substantial amount of mechanical puck-to-puck coupling, which limits the ability of the array to create spatially localized and distributed effects. A first psychophysical experiment showed that PixeLite's sensations can be localized to an area of about 30% of the total array area. A second experiment, however, showed that exciting neighboring pucks out of phase with one another in a checkerboard pattern did not generate perceived relative motion. Instead, mechanical coupling dominates the motion, resulting in a single frequency felt by the bulk of the finger.

Inducing Self-Motion Sensations With Haptic Feedback: State-of-The-Art and Perspectives on "Haptic Motion".

While virtual reality applications flourish, there is a growing need for technological solutions to induce compelling self-motion, as an alternative to cumbersome motion platforms. Haptic devices target the sense of touch, yet more and more researchers managed to address the sense of motion by means of specific and localized haptic stimulations. This innovative approach constitutes a specific paradigm that can be called "haptic motion". This article aims to introduce, formalize, survey and discuss this relatively new research field. First, we summarize some core concepts of self-motion perception, and propose a definition of the haptic motion approach based on three criteria. Then, we present a summary of existing related literature, from which we formulate and discuss three research problems that we estimate key for the development of the field: the rationale to design a proper haptic stimulus, the methods to evaluate and characterize self-motion sensations, and the usage of multimodal motion cues.

Study of the Effectiveness of a Wearable Haptic Interface With Cutaneous and Vibrotactile Feedback for VR-Based Teleoperation.

The control principle of teleoperated robotic systems is often not very intuitive, resulting in substantial cognitive load for the operator. Besides, the teleoperation of robotic systems requires stable and convenient visual feedback for the operator. Better spatio-temporal robot coordination can be achieved by providing the operator with high-fidelity multimodal haptic feedback. Using the advantages of virtual reality (VR) and haptic technology, we can improve the quality of teleoperation and increase the level of user involvement when operating the system. This article focuses on studying the effectiveness of the developed wearable haptic interface, vDeltaGlove, that provides multimodal haptic feedback when controlling a robotic arm through a virtual environment (VE). We evaluated the performance of vDeltaGlove in the series of teleoperation tasks. vDeltaGlove interface showed comparable characteristics with the Omega.7 desktop haptic device in terms of accuracy and outreached the performance of operation with VIVE controller. Besides, the developed interface showed the superior evaluation in terms of user experience. The proposed technology suggests a novel way of human-robot interaction (HRI) to achieve intuitive and immersive robot control.

HAPmini: 2D haptic feedback generation using single actuator device.

This study aims to explore a feasible form of a haptic device for common users. We propose HAPmini, a novel graspable haptic device that enhances the user's touch interaction. To achieve this enhancement, the HAPmini is designed with low mechanical complexity, few actuators, and a simple structure, while still providing force and tactile feedback to users. Despite having a single solenoid-magnet actuator and a simple structure, the HAPmini can provide haptic feedback corresponding to a user's 2-dimensional touch interaction. Based on the force and tactile feedback, the hardware magnetic snap function and virtual texture were developed. The hardware magnetic snap function helped users perform pointing tasks by applying an external force to their fingers to enhance their touch interaction performance. The virtual texture simulated the surface texture of a specific material through vibration and delivered a haptic sensation to users. In this study, five virtual textures (i.e., reproductions of the textures of paper, jean, wood, sandpaper, and cardboard) were designed for HAPmini. Both HAPmini functions were tested in three experiments. First, a comparative experiment was conducted, and it was confirmed that the hardware magnetic snap function could increase the performance of pointing tasks to the same extent as the software magnetic snap function could, which is commonly used in graphical tools. Second, ABX and matching tests were conducted to determine whether HAPmini could generate the five virtual textures, which were designed differently and sufficiently well for the participants to be distinguished from each other. The correctness rates of the ABX and the matching tests were 97.3% and 93.3%, respectively. The results confirmed that the participants could distinguish the virtual textures generated using HAPmini. The experiments indicate that HAPmini enhances the usability of touch interaction (hardware magnetic snap function) and also provides additional texture information that was previously unavailable on the touchscreen (virtual texture).